The factorization of state-action value functions for Multi-Agent Reinforcement Learning (MARL) is important. Existing studies are limited by their representation capability, sample efficiency, and approximation error. To address these challenges, we propose, ResQ, a MARL value function factorization method, which can find the optimal joint policy for any state-action value function through residual functions. ResQ masks some state-action value pairs from a joint state-action value function, which is transformed as the sum of a main function and a residual function. ResQ can be used with mean-value and stochastic-value RL. We theoretically show that ResQ can satisfy both the individual global max (IGM) and the distributional IGM principle without representation limitations. Through experiments on matrix games, the predator-prey, and StarCraft benchmarks, we show that ResQ can obtain better results than multiple expected/stochastic value factorization methods.

The factorization of state-action value functions for Multi-Agent Reinforcement Learning (MARL) is important. Existing studies are limited by their representation capability, sample efficiency, and approximation error. To address these challenges, we propose, ResQ, a MARL value function factorization method, which can find the optimal joint policy for any state-action value function through residual functions. ResQ masks some state-action value pairs from a joint state-action value function, which is transformed as the sum of a main function and a residual function. ResQ can be used with mean-value and stochastic-value RL.

Yet the combination of these factors created a milestone in AI history, as it had a profound impact on real-world applications and the successful deployment of various AI techniques that have been in the works for a very long time, particularly neural networks.g I shared these remarks in various contexts during the course of preparing this article. The audiences ranged from AI and computer science to law and public-policy researchers with an interest in AI. What I found striking is the great interest in this discussion and the comfort, if not general agreement, with the remarks I made. I did get a few "I beg to differ" responses though, all centering on recent advancements relating to optimizing functions, which are key to the successful training of neural networks (such as results on stochastic gradient descent, dropouts, and new activation functions). The objections stemmed from not having named them as breakthroughs (in AI). My answer: They all fall under the enabler I outlined earlier: "increasingly sophisticated statistical and optimization techniques for fitting functions." Follow up question: Does it matter that they are statistical and optimization techniques, as opposed to classical AI techniques?

The recent success of neural networks in applications such as speech recognition, vision and autonomous navigation has led to great excitement by members of the artificial intelligence (AI) community and the general public at large. Over a relatively short period, by the science clock, we managed to automate some tasks that have defied us for decades and using one of the more classical techniques coming out of artificial intelligence research. The triumph over these achievements has led some to describe the automation of these tasks as having reached human level intelligence. This perception, originally hinted at in academic circles, has been gaining momentum more broadly and is leading to some implications. For example, a trend is emerging in which machine learning research is being streamlined into neural network research, under its newly acquired label "deep learning." This perception has also caused some to question the wisdom of continuing to invest in other machine learning approaches, or even mainstream areas of artificial intelligence, such as knowledge representation, symbolic reasoning and planning. Some coverage of AI in public arenas, particularly comments made by some visible figures, has led to mixing this excitement with fear of what AI may be bringing us in the future (i.e., doomsday scenarios).